Skip to content

fix(migrate-apim): stop the planner silently losing policies - #66

Open
Facundo Santiago (santiagxf) wants to merge 5 commits into
mainfrom
fix-policy-planning-gaps
Open

Facundo Santiago (santiagxf) wants to merge 5 commits into
mainfrom
fix-policy-planning-gaps

Conversation

@santiagxf

@santiagxf Facundo Santiago (santiagxf) commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Closes #58, closes #59, closes #60, closes #65.

The planner could lose a policy four different ways:

  • dropped because no rule answered for it
  • stranded on a fragment it couldn't trace to an API
  • skipped before anything read it
  • marked unappliable and then never checked.

In every case the plan still looked fine. This fixes all four.

#58 — resolved in _apis_in_scope() by walking fragmentRefs backwards, rather than expanding a fragment into one batch member per including API, which is the other option the issue suggests.

#59 — the verdict check moved after _apply_partial_when(), and partialWhen learned hasExpression and underConditional. The trigger is rule data rather than element names in the planner, so authentication-managed-identity and a plain <set-backend-service backend-id="pool"/> behave exactly as before.

#65 — three places, because none of them is enough alone: a terminal rule in rules.json so None can't happen at runtime, a compile check so that rule can't quietly become the mapping, and a coverage tally so the planner can be checked against the inventory instead of trusted. The ten unanswered elements are covered by five family rules and one element rule, so the next member of each family is answered before anyone writes it.

#60 — blocking, and it reads from steps_for() rather than re-deriving which items become steps. Two copies of that predicate drifted once already, and an object no phase will run can't fail a phase — blocking on it would stop a migration over work you were going to do by hand anyway.

One test spans them. #60 is the safety net for the other three, and each half passes its own tests with the other absent, so that join was untested. Now: a fragment an API includes raises nothing to block on, and a fragment nothing includes stops at preflight naming the fragment, instead of taking out the phase.

Testing

583 passed, 1 skipped, compile --strict clean.

All offline — no Azure resources created, nothing run against a live APIM instance or gateway. These are contract bugs, decidable against the committed catalog, the CLI surface snapshot and the fake ARM fixture. The export job covers the other half by installing the pinned wheel.

Not fixed here

#63 — nothing checks that the partialWhen conditions rules.schema.json accepts are the ones _apply_partial_when() actually implements. It's already on main, but #59 adds two conditions to that vocabulary, so this PR widens the gap. Fix it here or follow up?

🤖 Generated with Claude Code

Factoring one `llm-token-limit` into a fragment and including it from several
APIs is how APIM shares a policy, and it was the one shape the planner could
not place. A fragment is its own document in the inventory, so its elements
were classified once, scoped to the fragment, and `_apis_in_scope` had no case
for `fragment`: it fell through to the empty set, no model registration
matched, and the item shipped as ready carrying an object with no model on it.
At apply time that object raises on the missing `modelName` and the policies
phase breaks out of its step loop, skipping every policy behind it. Re-planning
could not help -- it produces the same object -- and the reason it gave was
wrong as well as unhelpful: for a fragment there is something to carry the
policy, the planner just had no path from it to the including APIs.

The path is the `fragmentRefs` the collector already records from
`<include-fragment>`, read backwards. Each including scope is then resolved as
itself, so a fragment included globally covers every model and one included by
two APIs covers both, and a fragment included by a fragment is followed through.

Resolved in `_apis_in_scope` rather than by expanding the fragment into one
batch member per including scope, which was the other way to do it. A member
is a source policy, and the fragment is one source policy however many APIs
include it: expanding it would have reported four occurrences of a limit
written once, inflated the variants a gateway-wide baseline records collapsing,
and left each member naming an API scope while pointing at the fragment's
document, which is where `ip-filter` has to find its address ranges. The
baseline path never consulted API scope and still does not.

A fragment nothing includes keeps the stranded object: unreferenced, it governs
nothing, and that is the genuinely empty scope the message was written for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`plan` sets `configurationComplete: false` with an `incompleteReason` on an
object it cannot shape in full, and nothing read either field back. The
condition surfaced only when the step ran: the builder refused, reporting
whatever cause it could work out, and a failed step stops its phase -- so a
problem the planner had already detected and explained cost the rest of a
phase to rediscover, in worse words.

Preflight now enumerates them. Blocking, because the builder cannot produce
the object either and the run stops at it either way, which is the bargain
`plannerGap` already makes. Scoped to `steps_for`, so the objects checked are
the objects that run: one on work no phase performs cannot fail a phase, and
blocking on it would stop a migration over work the operator was going to
apply by hand, which `notExecutable` already names. The message quotes
`incompleteReason` and `fullConfigurationAt` rather than restating them,
since the planner is the only thing that knows why; two planner paths mark an
object incomplete from empty settings alone and write no reason, and those
report the silence instead of an empty quote.

The three fields are a contract between `plan` and `apply` and now a blocking
one, so the plan schema declares them. The rest of an object stays open,
because its shape is the capability's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`handled` is the strongest claim the planner makes about a policy element:
another object carries this, so there is nothing to decide. `_policies` acted
on it before `_apply_partial_when` ran, so it was a claim about the element
name rather than about the element.

`set-backend-service` is where that fails. The rule says "Backend selection
becomes the alias backend pool", and the pool is built from the APIM backend
pool, which carries no conditional logic at all. So
`<set-backend-service backend-id="@(expression)" />`, and the same element
inside a `choose`, were classified and then dropped: no item, no gap, and
nothing in the plan recording that routing logic had existed. The same
project rule that `collect` answered when a backend id built inside a policy
expression was invisible -- a failed inference must not read as a confident
negative -- was unanswered here one level up.

Analyse first, exclude after. `partialWhen` already expresses "this mapping
is lossy for this element" in data, so it gains two conditions beside
`attributeIsExpression`: `hasExpression`, and `underConditional`, read off
the path, which is the only place the enclosing `choose` survives
normalization. Any one of them demotes; requiring all would let an element
that trips a single condition read as cleanly mapped.

So the routing becomes a `partial` / `accept-loss` item proposing
`compensate`, carrying `alias.create` so the plan says what did travel, with
accept, compensate and retain-in-apim spelled out. Not a `gap`: the backends
and their priority and weight do reach the alias, and blocking a migration
over a `choose` between two members of one pool is the false alarm that
teaches people to ignore the plan. Not a note either -- `needs-decision` is
this plan's word for a fact somebody has to dispose of, and anything weaker
is not counted, not carried across a regeneration, and not read by `apply`.

Nothing changes for a `handled` element that is plain and unconditional, or
for one whose rule declares no condition: `authentication-managed-identity`
is a sound `handled` -- the provider item genuinely carries it -- and is
still excluded, expression or no expression. The trigger is per-rule data,
not a blanket policy, so plans do not fill with items for mappings that
really did hold.

The schema closes `partialWhen` and the `partial` overlay. A condition key
the planner does not know would never fire while reading as though it did,
which is the defect `additionalProperties: false` is there to catch.

559 passed, 1 skipped. compile --strict exits 0; no selector changed, since
`partial` and `partialWhen` were already in the policy sites' `reads`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three fixes land as one chain and nothing tested the joins. A policy
factored into a fragment now reaches the models of every including API, and the
one case left over -- a fragment nothing includes, which governs nothing -- is
caught by preflight in the planner's own words instead of failing a step and
skipping every policy behind it. Pinned where the two meet, because each half
passes its own tests with the other half absent.

The preflight section listed the conditions `--preflight` refuses on and was
three short, two of them from earlier work. A list that reads as complete and
is not teaches an operator to trust it.

Also drops a `label` in `_policies` that nothing has read since the item stopped
carrying one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_policies` dropped any element `policy_rule` did not answer for, and it did
not answer for ten of them: authentication-basic, authentication-certificate
and get-authorization-context, which carry backend credentials; check-header
and validate-client-certificate, which gate callers; emit-metric and
log-to-eventhub; mcp-server and validate-mcp-request; set-service-url; and
every policy APIM ships from now on. None of them reached the plan in any form.
An `<authentication-basic>` was read, classified, and discarded in silence.

The selection order made it worse than it reads. An element carrying a policy
expression fell into the hasExpression pass and was reported as a gap, so what
vanished completely were the elements with no expression in them -- the simpler
the policy, the more certainly it disappeared.

Closed in the three places it can be closed, because no one of them is enough.

A terminal rule, tried after the expression pass and matching anything, makes
"no rule" unreachable at runtime; what reaches it is a blocking gap, since the
alternative is migrating past a policy nobody has read. It is data, not a
default in the planner, so what an unknown element costs stays editable.

A net makes a poor mapping, so `compile` holds every element in policy.FAMILIES
to a rule of its own or a family rule and fails naming any left to the terminal
rule. FAMILIES is committed data, so the question is decidable offline, which is
what compile is for. The eleven are answered by five family rules and one
element rule rather than eleven entries, so the next member of a family is
answered before it is written.

And the plan now counts what became of every element walked, under exactly one
disposition each, with the unrecognised ones named and located. Three
dispositions raise no item; while they were uncounted, a plan that had
classified an element and discarded it could not be told from one that had never
seen it. `walked` is the sum, so the planner can be checked against the
inventory instead of trusted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@santiagxf Facundo Santiago (santiagxf) changed the title migrate-apim: stop a policy element being dropped, stranded, or excluded before it is read migrate-apim: fix four ways a policy could go missing during planning Nov 17, 2026
@santiagxf Facundo Santiago (santiagxf) changed the title migrate-apim: fix four ways a policy could go missing during planning migrate-apim: stop the planner silently losing policies Nov 17, 2026
@santiagxf Facundo Santiago (santiagxf) changed the title migrate-apim: stop the planner silently losing policies fix(migrate-apim): stop the planner silently losing policies Nov 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical planner defects can silently omit conditional routing, misapply token limits across workloads, and preserve saved decisions even when governing predicates or effective inclusion scopes change.

13 open issues
What changed in this PR

Hardens the migrate-apim workflow for APIM-to-AI Gateway migrations by making policy treatment visible and checking incomplete executable objects before apply.

Changes:

  • Resolves fragment policies to including APIs and surfaces conditional routing as migration decisions.
  • Adds fallback mappings, policy-vocabulary checks, and per-element coverage accounting.
  • Adds incomplete-object preflight checks, with supporting schemas, tests, and documentation.
File Description
tools/​migrate-apim/​tests/​test_plan.py Adds routing, coverage, and fragment-scope regression cases.
tools/​migrate-apim/​tests/​test_compile.py Tests vocabulary coverage and terminal-rule checks.
tools/​migrate-apim/​tests/​test_capability.py Updates fallback and known-policy selection tests.
tools/​migrate-apim/​tests/​test_apply.py Tests incomplete-object blocking and fragment integration.
tools/​migrate-apim/​schemas/​rules.schema.json Defines conditional overlays and terminal-rule matching.
tools/​migrate-apim/​schemas/​plan.schema.json Describes coverage and object-completeness fields.
tools/​migrate-apim/​README.md Documents mappings, fragment handling, coverage, and preflight.
tools/​migrate-apim/​migrate_apim/​plan.py Tracks policy dispositions and resolves fragment includers.
tools/​migrate-apim/​migrate_apim/​compile.py Checks policy-vocabulary coverage and terminal-rule presence.
tools/​migrate-apim/​migrate_apim/​capability.py Shares four-pass policy selection with compilation.
tools/​migrate-apim/​migrate_apim/​apply.py Reports and blocks executable objects marked incomplete.
tools/​migrate-apim/​capabilities/​rules.json Adds missing mappings, fallback coverage, and routing-loss decisions.
tools/​migrate-apim/​ARCHITECTURE.md Explains four-pass selection and policy coverage.

Review effort: Max

Comment on lines +371 to +372
"value": "compensate",
"consequence": "Say what the condition was for in the priority and weight of the alias backend models, as far as it can be said that way."
Comment on lines +950 to 955
if not rule:
# Unreachable while rules.json carries a terminal rule, and
# counted rather than deleted because that is a property of
# the data, which `compile` enforces and this does not.
self._count_element("unresolved", document, element)
continue
Comment on lines +1510 to +1513
for include_level, include_name in _fragment_includers(
name or "", self.inventory.get("policies") or []
):
apis = self._apis_in_scope(include_level, include_name)
Comment on lines +1510 to +1513
for include_level, include_name in _fragment_includers(
name or "", self.inventory.get("policies") or []
):
apis = self._apis_in_scope(include_level, include_name)
Comment on lines +1513 to +1515
apis = self._apis_in_scope(include_level, include_name)
if apis is None:
return None
holds = (
(isinstance(value, str) and value.lstrip().startswith("@"))
or (condition.get("hasExpression") and bool(element.get("expressionIds")))
or (condition.get("underConditional") and _under_conditional(element))
Comment on lines +2106 to +2109
by_fragment: dict[str, list[dict[str, Any]]] = {}
for document in documents:
for name in document.get("fragmentRefs") or []:
by_fragment.setdefault(name, []).append(document.get("scope") or {})
Comment on lines +290 to +294
"match": {
"family": "authentication"
},
"verdict": "gap",
"itemKind": "accept-loss",
Comment on lines +272 to +274
"Re-plan from a current inventory, or exclude the items named above to leave their "
"objects out of the run. Applied as it stands, each one fails its step and stops the "
"rest of its phase.",
seen.add(nested)
pending.append(nested)
continue
entry = (scope.get("level"), scope.get("name"))

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants